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Why  T rust  Matters 
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■  Congressional  Mandate 

■  The  National  Defense  Authorization  Act  for  Fiscal 
Year  2001,  Public  Law  106-398,  Congress 
mandated  in  Section  220  that  “It  shall  be  a  goal  of 
the  Armed  Forces  to  achieve  the  fielding  of 
unmanned,  remotely  controlled  technology  such 
that...  by  2015,  one-third  of  the  operational  ground 
combat  vehicles  are  unmanned.”1 
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Why  Trust  Matters 
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■  Secretary  of  Defense  Gates’  Comments 

■  On  21  April  2008,  Secretary  Gates  made  the  following 
comment  about  unmanned  systems 

“Unmanned  systems  cost  much  less  and  offer  greater  loiter 
times  than  their  manned  counterparts,  making  them  ideal  for 
many  of  today's  tasks.  Today,  we  now  have  more  than  5,000 
UAVs,  a  25-fold  increase  since  2001.  But  in  my  view,  we  can 
do  and  we  should  do  more  to  meet  the  needs  of  men  and 
women  fighting  in  the  current  conflicts  while  their  outcome 
may  still  be  in  doubt.  My  concern  is  that  our  services  are 
still  not  moving  aggressively  in  wartime  to  provide 
resources  needed  now  on  the  battlefield.  I've  been  wrestling 
for  months  to  get  more  intelligence,  surveillance  and 
reconnaissance  assets  into  the  theater.  Because  people 
were  stuck  in  old  ways  of  doing  business,  it's  been  like 
pulling  teeth. ,J2 
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Why  Trust  Matters 
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■  P.W.  Singer  of  the  Brookings  Institution  stated  the 
following  in  an  article  published  this  year 

■  “So,  despite  what  one  article  called  ‘all  the  lip  service  paid  to 
keeping  a  human  in  the  loop,  ’  the  cold,  hard,  metallic  reality 
is  that  autonomous  armed  robots  are  coming  to  war.  They 
simply  make  too  much  sense  to  the  people  that  matter.”  4 
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Autonomy  Defined 
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■  The  American  Heritage  Dictionary  defines 
autonomy  as 

m  “Independence.  ”  5 

■  Wikipedia  defines  an  autonomous  agent  as 

■  “A  system  situated  in,  and  part  of,  an  environment, 
which  senses  that  environment,  and  acts  on  it,  over 
time,  in  pursuit  of  its  own  agenda.  This  agenda  evolves 
from  drives  (or  programmed  goals).  The  agent  acts  to 
change  the  environment  and  influences  what  it  senses 
at  a  later  time.  ” 

■  “Non-biological  examples  include  intelligent  agents, 
autonomous  robots,  and  various  software  agents, 
including  artificial  life  agents,  and  many  computer 
viruses.  ”  6 
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Autonomy  Defined 
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■  There  are  varying  degrees  of  autonomy 

■  None  -  System  is  completely  manually  controlled 

■  Partial  -  Some  functions  are  automated 

■  Sliding  -  The  amount  of  autonomy  is  selectable 

■  Full  -  The  system  operates  entirely  without  human  control 

■  With  any  autonomy  the  user  gives  up  some  level  of 
control 

■  Wikipedia  has  a  good  definition  of  the  abilities  of  a  fully 
autonomous  robot 

■  " A  fully  autonomous  robot  has  the  ability  to 

•  Gain  information  about  the  environment 

•  Work  for  an  extended  period  without  human  intervention 

•  Move  either  all  or  part  of  itself  throughout  its  operating 
environment  without  human  assistance 

•  Avoid  situations  that  are  harmful  to  people ,  property,  or  itself 
unless  those  are  part  of  its  design  specifications.” ' 
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Autonomy  Defined 
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Examples  of  Degrees  of  Autonomy 


Partial  Autonomy 
No  Autonomy  (ISS  assembly 

(Remote-control  robot) 

EOD  robot) 


Sliding  Autonomy 
(UAS) 


Full  Autonomy 
(ALCM) 


Boeing  Photo 


US  Air  Force  Photo 


NASA  Photo 


US  Air  Force  Photo 
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Trust  Defined 
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■  The  “classic”  definition  of  trust  comes  from 
Diego  Gambetta 

■  “trust  (or,  symmetrically,  distrust)  is  a  particular  level 
of  the  subjective  probability  with  which  an  agent 
assesses  that  another  agent  or  group  of  agents  will 
perform  a  particular  action,  both  before  he  can 
monitor  such  action  (or  independently  of  his  capacity 
ever  to  be  able  to  monitor  it)  and  in  a  context  in  which 
it  affects  his  own  action.  When  we  say  we  trust 
someone  or  that  someone  is  trustworthy,  we  implicitly 
mean  that  the  probability  that  he  will  perform  an  action 
that  is  beneficial  or  at  least  not  detrimental  to  us  is 
high  enough  for  us  to  consider  engaging  in  some  form 
of  cooperation  with  him.”8 
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Research  Objective 
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■  Assuming  that  autonomous  agents  are  coming  to 
hazardous  environments,  like  disaster  areas  and 
combat  zones,  how  can  we  trust  them? 

■  This  research  project  is  attempting  to  identify  the 
factors  that  contribute  to  trust  in  autonomous 
agents,  in  order  to  develop  a  set  of  key 
performance  parameters  (KPPs)  for  trusted 
agents 

■  These  KPPs  can  be  used  by  intelligent  agent 
developers  to  validate  the  trustworthiness  of  their 
agents  and  to  convince  users  of  their 
trustworthiness 
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Research  Objective 
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■  Three  elements  to  the  research 

■  Identify  factors  of  trust  published  in  the  literature 

■  Simulate  human-robot  interactions 

■  Collect  data  on  human  interactions  with 
autonomous  agents 

■  The  ultimate  objective  is  to  develop  a  process 
that  can  be  validated  in  field  trials 
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Identifying  Trust  Factors 
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■  Cristiano  Castelfranchi  and  Rino  Falcone  argue  that 

Only  agents  endowed  with  goals  and  beliefs 
(cognitive  agents)  can  “trust"  another  agent9 

■  They  define  the  elements  of  trust  with  the  following 
diagram9 


GOAL  g 

Bi:  y  can  g,  has  the  power  of  g  (Evaluation) 

B2:  y  will-do  a  for  g  (Expectation) 

B3:  g  will  be  true  (Trust  that  g)  CORE  TRUST 


B4: 1  need  y  for  g  (Dependence) 

GOAL  of  not  doing/  not  exploit  alternatives/ 
beting  on  y  (Reliance  and  bet) 

GOAL  that  y  can  &  will  do _ RELIANCE^ 

Mental  ingredients  of  TRUST 
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Identifying  Trust  Factors 
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■  Sarvapali  Ramchurn,  et  al.,  identified  two 
principle  components  of  trust  in  an  agent10 

■  Confidence  -  Do  I  believe  the  agent  can  perform 
the  desired  task? 

■  Reputation  -  Has  this  agent  been  successful  in 
the  past  and  have  others  trusted  this  agent,  with 
good  results? 
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Identifying  Trust  Factors 
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■  Karen  Fullam  and  Suzanne  Barber  focused  on 
the  importance  of  reputation  when  dealing  with 
agents  (either  human  or  artificial)  in  the 
development  of  the  ART  Testbed11 
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Identifying  Trust  Factors 
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■  Just  in  these  three  research  papers,  the  following 
potential  trust  factors  were  identified 

■  Evaluation 

■  Expectation 

■  T  rust 

■  Dependence 

■  Reliance 

■  Bet 

■  Confidence 

■  Reputation 

■  Which  factors  are  the  key  factors? 
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Simulating  Trust 
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■  Trust  of  autonomous  systems  is  an  active  area  of 
research 

■  Much  of  the  research  uses  simulations  to  explore 
issues  of  trust  and  to  compare  approaches 

■  The  simulations  tend  to  take  two  forms 

■  Simulations  designed  specifically  for  examining 
issues  of  trust 

■  Simulations  that  were  designed  for  other  areas  of 
research  that  have  been  extended  or  adapted  to 
trust  research 

■  The  following  are  some  examples  of  both  types 
of  simulation 


20  April  2009 


Systems  &  Software  Technology  Conference  2009 


17 


Simulating  Trust 
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■  Agent  Reputation  and  Trust  (ART)  Testbed12 

■  A  simulation  of  multiple  art  appraising  agents 
specifically  designed  for  agent  trust  research 


20  April  2009 


Systems  &  Software  Technology  Conference  2009 


18 


Simulating  Trust 
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RoboCup  Rescue  Simulation13 

■  An  agent  simulation  system  that  has  been  used 
for  trust  research14 


Coalition  Formation 


mm 

iL_  %^A 

Estimation  with 
Faulty  Sensors 


Resource  Allocation 
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■ 


Simulating  Trust 
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■  Firefighting  Simulation 

■  A  simulation  of  human-robot  trust  relationships 
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Human-Robot  Trust  Game 

Prototype 
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■  A  prototype  of  a 
game  for 
collecting  user 
interaction  data 
was  created  in 
NetLogo 

■  The  objective  of 
the  prototype 
was  to  evaluate 
playability  and 
design  elements 
for  data 
collection 
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Web-based  Game 
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■  To  collect  data  from  a  variety  of  users,  a  web- 
based  game  is  being  developed  using  the  Google 
Web  Toolkit,  which  provides  a  variety  of  tools  for 
collecting  data  on  the  backend  of  a  website 

■  The  project  page,  with  the  latest  version  of  the 
game  and  supporting  material,  can  be  found  at 


https://sites.google.eom/a/aggiemail.usu.edu/ 

h  u  m  a  n -ro  bot-t  r  u  st-p  roj  ect/ 
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Identifying 

Human-Agent  Trust  KPPs 
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■  The  future  direction  for  this  project  is  to  develop 
Key  Performance  Parameters  that  can  be  used  by 
autonomous  agent  designers 

■  Serve  as  design  guidelines 

■  Provide  parameters  for  testing  against  during 
development 

■  Provide  validation  parameters  for  field  testing 
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Trust  in  Reverse? 
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A  project  by  Kacie  Kinzer: 
http://www.tweenbots.com 
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Trust  in  Reverse? 
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Questions? 
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■  ALCM 

■  ART 

■ DARPA 

■  EOD 

■  ISS 

■  KPP 

■  LOAC 

■  NIST 

■  OSD 

■  SMXS 

■  UAS 

■  UAV 

■  UGV 

■  USV 


Acronym  List 
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Air  Launched  Cruise  Missile 

Agent  Reputation  and  Trust 

Defense  Advanced  Research  Projects  Agency 

Explosive  Ordnance  Disposal 

International  Space  Station 

Key  Performance  Parameter 

Law  Of  Armed  Conflict 

National  Institute  for  Standards  and  Technology 

Office  of  the  Secretary  of  Defense 

Software  Maintenance  Squadron 

Unmanned  Aerial  System 

Unmanned  Aerial  Vehicle 

Unmanned  Ground  Vehicle 

Unmanned  Submersible/Surface  Vehicle 
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